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ABSTRACT 

A  review  of  several  optimality  criteria  for  the  design  of  experiments 
involving  quantitative  factors  for  the  estimation  of  unknown  papameters  in 
regression  problems  is  given.  These  include  rotatability,  D-optimality  and  G- 
optimality  criteria.  Designs  are  no  longer  optimal  under  these  criteria  when 
conditions  are  suboptimal. 

The  two  main  lines  of  research  in  the  robust  design  of  experiments  will 
be  discussed,  i.e.  <i)  the  comparison  of  the  performance  of  an  optimal  design 
with  respect  to  a  particular  criterion  under  other  criteria;  (ii)  the 
construction  of  designs  which  guard  against  particular  short  comings. 
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SIGNIFICANCE  AND  EXPLANATION 


A  review  of  several  optimality  criteria  for  the  design  of  experiments 
involving  quantitative  factors  for  the  estimation  of  unknown  parameters  in 
regression  problems  is  given.  These  include:  rotatability,  the  variance  of 
the  estimated  response  at  a  particular  point  being  a  function  of  the  distance 
of  the  point  from  the  center  of  the  design:  D-optimality,  the  minimization  of 
the  generalized  variance;  and  G-optiraality ,  the  minimization  of  the  maximum 
variance  of  the  estimated  response.  Designs  are  no  longer  optimal  under  these 
criteria  when  conditions  are  suboptimal;  for  example  when  the  assumptions 
about  the  model  or  the  region  do  not  hold,  some  observations  are  missing,  or 
some  observations  are  outliers  and  need  to  be  set  aside. 

The  two  main  lines  of  research  in  the  robust  design  of  experiments  are 
discussed,  i.e.  (i)  the  comparison  of  the  performance  of  an  optimal  design 
with  respect  to  a  particular  criterion  under  other  criteria;  (ii)  the 
construction  of  designs  which  quard  against  particular  short  comings.  Robust 
designs  will  be  slightly  less  than  optimal  under  ideal  conditions  but  will  be 
more  efficient  under  more  realistic  conditions. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  paper. 


THE  ROBUST  DESIGN  OF  EXPERIMENTS:  A  REVIEW 


*f* 

Agnes  M.  Rerzberg 

1.  Introduction 

The  term  robust  was  Introduced  into  statistics  by  Box  (1953)  with  regard 
to  statistical  tests  being  valid  when  some  of  the  assumptions  did  not  hold. 
Since  1953,  there  has  been  a  large  proliferation  of  papers  on  the  subject  of 
robustness.  Andrews,  Bickel,  Hampel,  Huber,  Rogers  and  Tukey  (1972)  state: 


'Estimation  is  the  art  of  inferring  information 
about  some  unknown  quantity  on  the  basis  of  available 
data.  Typically  an  estimator  of  some  sort  is  used.  The 
estimator  is  chosen  to  perform  well  under  conditions  that 
are  assumed  to  under ly  the  data.  Since  these  conditions 
are  never  known  exactly,  the  estimators  must  be  chosen 
which  are  robust,  which  perform  well  under  a  variety  of 
underlying  conditions. 

The  theory  of  robust  estimation  is  based  on  the 
specified  properties  of  specified  estimators  under 
specified  conditions.' 

One  wants  to  determine  the  interactions  of  these. 

There  are  broadly  two  senses  in  which  robust  is  used  in  the  design  of 
experiments.  The  first  is  that  after  one  has  obtained  an  optimal  design 
relative  to  a  particular  criterion  one  examines  how  well  the  design  performs 
in  other  respects;  see,  for  example,  Kiefer  (1975),  Galil  and  Kiefer  (1977 
a,b)  and  Kiefer  and  Studden  (1976).  If  there  is  still  some  freedom  in  the 
construction  of  the  design,  one  can  incorporate  further  subsidiary  optimality 
requirements;  see  Herzberg  and  Cox  (1972).  The  other  sense  is  the 
construction  of  designs  which  guard  against  particular  short  comings;  see,  for 
example.  Box  and  Draper  (1975)  and  Herzberg  and  Andrews  (1976).  These  robust 
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designs  will  be  slightly  less  than  optimal  under  ideal  conditions  but  will  be 
more  efficient  under  more  realistic  conditions. 

Hedayat  and  John  (1974)  and  John  (1976)  have  discussed  robustness  for 
balanced  incomplete  block  designs.  Although  very  important,  this  work  will 
not  be  discussed  here. 

2.  General  Comments 

The  design  of  experiments  involving  quantitative  factors  has  been 
extensively  investigated,  two  main  lines  of  work  being  that  on  response 
surfaces  and  that  on  general  studies  of  optimality  in  designs  for  estimating 
regression  coefficients;  see  for  example.  Box  and  Wilson  (1951),  Box  and 
Hunter  (1957)  and  Box  and  Draper  (1959)  for  the  first  and  Elfving  (1952), 
Chernoff  (1953),  Kiefer  (1959,  1961,  1962)  and  Kiefer  and  Wolfowitz  (1959)  for 
the  second. 

It  is  well  known  that  caution  is  necessary  in  formulating  these 
problems.  There  are  three  aspects  of  primary  importance: 

(i)  the  region,  R,  of  experimentation  must  be  specified.  In  many  cases, 
the  optimal  design  points  will  be  on  the  boundary  of  R; 

(ii)  the  type  of  model  to  be  fitted  has  a  major  effect  on  the  optimal 
design.  A  choice  of  the  most  complex  type  of  model  to  be  fitted  has, 
therefore,  to  be  made.  If  this  is  not  feasible,  the  most  sensible  design  is 
to  take  all  the  design  points  distinct  and  covering  the  region  R;  there  is 
some  loss  of  efficiency  if  in  fact  a  simple  model  is  adequate; 

(iii)  the  special  objectives  of  the  experiment  have  to  be  formulated. 

Two  quite  distinct  considerations  may  determine  R.  Physical  constraints 

may  make  it  impossible  or  undesirable  to  work  outside  certain  ranges;  or  it 
may  be  thought  that  the  behavior  of  the  observations  in  certain  regions  of  the 
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factor  space  is  of  little  interest  or  at  least  unrepresentative  of  the 
behavior  in  the  main  region  of  concern.  In  the  former  case  a  fairly  clearly 
defined  region,  often  rectanglular ,  is  available,  but  in  the  second  case 
definition  of  R  is  more  difficult.  Sometimes,  however,  one  tries  to  measure 
the  factors  in  units  such  that  a  spherical  R  is  reasonably  suitable.  This 
suggests  three  main  cases  according  as  R  is  cuboidal,  spherical,  and 
cylindrical.  The  first  two  cases  have  been  extensively  studied. 

Possible  objectives  of  the  experiment  are  numerous.  It  may  be  difficult 
to  be  very  specific  and  several  of  the  objectives  may  be  involved 
simultaneously.  The  main  ones  are: 

(a)  the  estimation  of  one  or  more  parameters: 

(b)  the  testing  of  the  adequacy  of  one  or  more  proposed  models,  as  a 
guide  to  the  selection  of  a  model; 

(c)  the  discrimination  among  alternative  models; 

(d)  the  estimation  of  the  detailed  form  of  a  complex  curve  or  function; 

(e)  the  estimation  of  the  position  of  and  the  response  at  a  maximum  (or 
minimum) ; 

(f)  the  estimation  of  the  line  of  steepest  ascent  usually  from  the  center 
of  the  design; 

(g)  the  estimation  of  the  properties  of  the  canonical  form  of  the  second 
degree  equation; 

(h)  the  estimation  fo  the  expected  response  at  one  or  more  points. 

If  the  points  are  inside  R,  this  is  a  problem  of  interpolation,  whereas  if 
the  points  are  outside  R  it  is  one  of  extrapolation? 

(i)  the  estimation  of  the  diffarences  of  the  responses  between  given 
points.  When  the  points  are  close  together,  there  is  a  connection  with  (f). 


In  addition  to  the  previous  three  central  aspects ,  there  are  a  number  of 


further  aspects  of  importance,  including 

( iv)  the  possible  presence  of  additional  qualitative  factors; 

(v)  the  sequential  or  nonsequential  character  of  the  experiment; 

(vi)  the  error  structure,  for  example  blocking,  two-way  control,  split- 
plot  arrangement,  etc,; 

(vii)  the  presence  of  additional  constraints,  such  as  the  need  to  have  a 
small  number  of  levels  of  certain  factors. 


3.  D-optlmal  and  G-optlmal  Designs 
Let 

E{y(x)  }  -  f'(x)0,  (1) 

■"w 

where  y(x)  is  an  observation  at  the  point  x  in  the  design  region, 
f'(x)  *  {f „(x),...,f  (x)}  and  0*  »  (0.,...,0  )  ,  the  vector  of  unknown 

~  ~  1—  p  ->»  —  i  p 

2 

parameters  to  be  determined  by  least  squares;  var{y(x)}  =*  o  and 

covtylx^),  y(x^)}  *  0  (i  j)  . 

Three  types  of  design  are  of  interest: 

(i)  discrete  designs,  n  points  x  ,...,x  ,  giving 

^  n 

n 

X  *  {f(x)  }  and  X'X  -  {  £  f.(x  )f  (x  )}; 

(ii)  design  measures,  £  ,  giving 

M<$)  -  (®1j  }# 

where 


m 


ij 


/  f  (x)f  (x)£(dx),  x  being  the  space  of  interest; 

xeX  3  ~ 
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(iii)  associated  design  measures#  £  ,  where  each  point 

n 

2 

x„,...,x  has  measure  or  weight  n  and  X'X  »  nM( 0  ). 

~1  -n  ~  -»  —  n 

The  well-known  equivalence  theorem  of  Kiefer  and  Wolfowitz  (1960)  states 
the  following: 

Theorem  The  following  assertions 

(i)  £  maximizes  |M(£)|, 

* 

^  minimizes 

max  d(x,0)  -  f*(x)M-1(£)f(x), 

X  ^  ^  ^  v 

(iii)  max  d(x, £  )  *  p 
are  equivalent. 

The  first  assertion  defines  D~optimal  designs,  |M(£)|  1  being 

*w 

proportional  to  the  generalized  variances;  the  second  assertation 
defines  G-optimal  designs,  the  d(.,.)  being  related  to  the  variance  of  the 
extimated  response.  For  design  measures,  the  two  criteria  are  equivalent, 
designs  satisfying  (ii)  being  easier  to  determine  because  of  (iii). 

Consider  the  following  example.  Let 

E{y(x)}  -  e0  +  e1*1  *  02x2  *  e1lXl2  ♦  o22x22  +  e^x^, 
and  let  the  design  region  be  the  square  with  vertices  (+1,  +1)  .  Consider 
the  following  three  designs.  Design  I  has  measure  of  0. 1458+  at  the 
vertices,  0.08015+  at  the  midpoints  of  the  edges,  and  0.0962+  at  the 
origin.  Design  II  consists  of  9  points,  one  at  each  vertex,  midpoint  of  the 
edges  and  at  the  origin.  Design  III  consists  of  13  points,  2  at  each  vertex, 
one  at  the  midpoint  of  each  edge  and  one  at  the  origin. 

For  Design  I,  |M, (£)|  ■  0.0399,  max  d(x, £)  ■  6  ;  for  Design  II, 

|M(£)|  -  0.0098,  max  d(x,£)  -  7.25;  for  Design  III,  |M(£)|  -  0.0113, 

*w 

max  d(x,£)  ■  6.88.  Design  I  is  the  D-optimal  and  G-optimal  design  measure. 
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but  since  the  measure  is  irrational  the  design  cannot  be  performed.  Design  I 
serves  as  a  guide.  It  can  be  seen  that  Design  III  is  better  than  Design  II 
and  attains  values  closest  to  those  of  the  optimal  design. 


4.  Rotatable  Designs 

From  ( 1 ) ,  the  least  squares  estimator  of  9  is 

0  -  (X'XJ-’x'Y, 


(2) 


where  X  ■  {f(x  )}  and  Y  is  the  nxl  vector  of  observations.  Then,  if 


y(x)  -  f ' (x) 6, 

*  v  <>/  <v  v 


var{y(x)  }  -  f  •  (x)  (X'X)"*^ (x)  a2, 

-%J  <V  -V  V  X 


(3) 


a  being  the  experimental  error  variance.  Equation  (3)  will  be  a  function 
of  the  distance  of  the  point  from  the  center  of  the  design,  i.e. 

var{y(x) }  -  w(  Ixl)  (4) 

if  and  only  if 

-i  -i 

(5) 


(X'X)"1  -  N'tX'X)"^, 


*>J  •>/ 


where  N  is  a  particular  type  of  orthogonal  matrix;  see  Box  and  Hunter 
(1957).  A  design  which  satisfies  (4)  or  (5) is  called  a  rotatable  design. 

Thus  the  variance  contours  of  a  rotatable  design  are  spherical  and  the 
variance  of  certain  0^'s  are  constant  no  matter  what  orientation  of  the 
design.  If  the  experimenter  is  not  certain  in  which  part  of  the  design  region 
he  will  be  interested,  spherical  variance  contours  appear  to  be  advisable; 
see,  for  example.  Box  and  Hunter  (1957).  These  designs  could  be  considered  as 
being  variance  robust. 


5.  Model  Robustness 

Box  and  Draper  (1959,  1963)  considered  designs  that  minimize  the  mean 
square  error  over  the  spherical  region  of  the  factor  space.  They  minimized. 
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over  the  spherical  region  R,  the  function 


J  =  /  E  {y<  x)  -  n(  x)  }2  dx 

0  R 

*  ^  /  var{y(x)  dx  +  —  /  [E{y(x)}  -  n(x)]2  dx, 

or  ci  R 

where  y(x)  is  the  least  squares  estimate  of  the  assumed  polynomial  model, 

n(x)  is  the  true  model,  ft  1  »  j  dx  ,  n  is  the  number  of  design  points  and 
2~  R 

a  is  the  experimental  error  variance.  The  size  of  J  arises  partly  from 
bias  due  to  the  use  of  a  polynomial  of  too  small  degree  and  partly  from  random 
errors.  When  bias  alone  is  considered,  the  bias  is  minimized  by  rotatable 
designs.  Draper  and  Lawrence  (1965,  1966)  examined  the  situation  when  the 
region  is  cuboidal. 

Kiefer  and  Studden  (1976)  compared  designs  for  situations  where  the  model 
is  of  the  form 


E{y(x)}  =  £  e.x*  , 

i=*0 

the  design  region  is  t-1,1]  and  the  optimal  design  measure,  £n*  has  n 

points  of  support.  They  discussed  optimal  n  point  design  measures  for 

interpolation  and  extrapolation  and  compared  these  to  the  limiting  case 

when  n  +  “  .  It  is  shown  that  in  some  situations  it  may  be  better  when  the 

model  is  of  order  n  to  use  5  where  m  >  n. 

in 

6.  The  Variance  of  the  Difference  Between  Two  Responses 

The  experimenter  may  not  always  be  interested  in  the  estimated  response 
at  one  point  but  changes  of  response,  for  example,  in  the  difference  between 


7 


V 


the  estimated  responses  at  two  points.  Herzberg  (1967)  considered  the 
behavior  of  the  variance  function  of  the  difference  between  two  estimated 
responses.  It  was  shown  that  for  rotatable  designs,  the  variance  function  of 
the  difference  between  two  estimated  responses  is  a  function  only  of  the 
distances  of  the  points  from  the  origin  of  the  design  and  the  angle  subtending 
them  at  the  origin.  Box  and  Draper  (1980)  have  extended  and  unified  these 
results. 

7.  Measures  of  Design  Robustness 

Herzberg  and  Andrews  (1976)  introduced  robust  measures  for  situations 
where  outliers,  missing  values  or  non-Gaussian  distributions  are  contemplated. 

Let  a(x)  be  the  probability  of  losing  an  observation  at  the  point  x  , 
the  losses  at  different  points  being  independent.  Let  D  -  {d^},  a 
diagonal  matrix,  where 

0  with  probability  crtx^), 

1  with  probability  1  -  afx^), 

the  value  0  being  associated  with  a  missing  observation.  This,  of  course, 
does  not  explain  missing  observations,  but  is  only  a  formulation  to 
investigate  their  potential  influence  on  experimental  design. 

If  a  sufficient  number  of  observations  are  missing,  not  all  of  the 
elements  of  6  given  in  (1)  may  be  estimated.  This  occurs  when 
|X'DXj  -  0  .  The  probability  that  this  occurs  is 

pr( |X’DX|  -  0)  (6) 

v  w 


which  may  be  used  to  compare  designs.  This  measure  is  called  the  probability 


of  breakdown  of  the  design.  Designs  robust  under  this  measure  have  a  small 
value  for  (6). 

Another  measure  of  robustness  is 

E( iX'DXl 1/p) ,  (7) 

which  is  related  to  the  D-optimality  criterion.  Designs  robust  under  this 
measure  have  (7)  large. 

Another  measure  of  robustness  and  some  examples  are  given  in  Herzberg  and 
Andrews  (1976).  Further  examples  and  computational  methods  are  given  in 
Andrews  and  Herzberg  (1979).  Herzberg  and  Andrews  (1978)  compare  the 
robustness  of  chain  block  and  coat-of-mail  designs. 

Box  and  Draper  (1975)  introduced  a  criterion  for  the  construction  of 
designs  to  minimize  the  effect  of  outliers  in  the  least  squares  estimate  of 
the  response  function.  For  the  comparison  of  designs,  the  value  of  the 
population  variance  of  the  variances  of  the  estimated  response  at  the  design 
points  is  used  implying  the  smaller  the  value  the  more  robust  the  design. 

8.  Robustness  Against  Autocorrelation  in  Time 

The  assumption  of 

cov {y( x  ) ,  y(x,)}  *  0  (i  *  j) 

-1  ~j 

is  very  often  artificial.  Bickel  and  Herzberg  (1979)  assume  that 

2 

cov{y(x1),  y(x,)]  -  v^o 

2 

and  the  variance-covariance  matrix  of  the  observations  is  Vo  ,  where 
V  =  {v^}.  They  find  one-dimensional  optimal  designs  for  particular 
situations  when  the  number  of  observations,  n  ,  tends  to  infinity  and  V  is 
unknown  but  of  an  assumed  form.  Examples  and  further  results  are  given  in 
Bickel,  Herzberg  and  Schilling  (1981). 
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9.  Conclusion 


A  review  of  several  optimality  criteria  for  the  design  of  experiments  has 
been  given.  Some  of  these  designs  possess  certain  robust  qualities.  Measures 
of  robustness  of  designs  are  introduced  in  order  that  designs  may  be 
compared.  It  is  very  important  also  in  this  connection  to  examine  the  results 
of  Kiefer  (197S)  and  Galil  and  Kiefer  ( 1977a, b)  where  optimal  designs  relative 
to  a  particular  procedure  are  examined  for  their  performance  with  respect  to 
other  criteria. 
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